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ABSTRACT 

This paper aims to identify the operational region of a link 
in terms of its utilization and alert operators at the point 
where the link becomes overloaded and requires a capacity 
upgrade. The number of active flows is considered the real 
network state and is proposed to use a proxy for utiliza- 
tion. The Gaussian approximation gives the expression for 
the confidence interval on an operational region. The easy 
rule has been formulated to display the network defects by 
means of measurements of router loading and number of ac- 
tive flows. Mean flow performance is considered as the ba- 
sic universal index characterized quality of network services 
provided to single user. 

Categories and Subject Descriptors 

C.2.3 [Computer-communication networks]: Network 
Operations — network monitoring; C.4 [Performance of sys- 
tems]: Measurement techniques 

Keywords 

Network States on Flow Level, Mean Flow Performance, 
Test for Network Quality 

1. INTRODUCTION 

Typically, the following four values are used for the estima- 
tion of the network quality: 
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• Link or router loading (or available bandwidth for end- 
to-end connection) 

• Round trip time 

• Packet loss rate 

• IP packet delay variation (network jitter) 

The available bandwidth, round trip time, packet loss rate 
and IP packet delay variation describe the quality of connec- 
tivity between two remote points or end-to-end connection. 
The link utilization is applied to the monitoring of a single 
hop between two routers [9] [16] . 

Network operators need to know when their backbone or 
peering links must be upgraded. Boundary values of network 
parameters may serve as an indicator e.g. as the current val- 
ues of the network parameters reach a defined limit, the links 
have to be upgraded. The problem with this method is that 
there is no standardized set of network parameters to moni- 
tor. Each provider has its own set of technical specifications 
aimed at avoiding overload. Big providers, like Sprint [9], 
rely on the results of their own research. Usually, network 
operators monitor peak and average link utilization levels 
and upgrade their links when the utilization level is in the 
range 30%-60%. 

The main focus of this paper is to use flow-based analy- 
sis [5] to monitor the backbone link and identify when an 
upgrade is needed. Previous work by Chuck Fraleigh et al [9] 
addressed a similar provisioning problem to reduce the per 
packet end-to-end delay. Dina Papagiannaki et al [l6] , at In- 
focom 03, introduced a methodology on the basis of SNMP 
statistics to predict when and where link additions/upgrades 
should take place in an IP backbone network. 



For resource management in IP networks, several recent pro- 
posals [6] advocate for a flow oriented architecture. A flow 
is classically identified by the usual 5tuple composed of the 
source and destination addresses together with the source 



and destination port numbers and protocol type. The ba- 
sic principles of flow-aware networking rely on the fact that 
flows are the elementary entities associated with user trans- 
actions. In order to provide end users with an acceptable 
quality, it is essential to share the bandwidth of the network 
by taking into account flows [4] |11|[T3]|15| . To efficiently im- 
plement flow aware resource management techniques, how- 
ever, it is essential to estimate the number of flows simulta- 
neously active on a link in an operational network [2] [8] . 

Traffic accounting mechanisms based on flows should be con- 
sidered as passive measurement mechanisms. Information is 
gathered by flows are useful for many purposes: 

• Understanding the behaviour of existing networks 

• Planning for network development and expansion 

• Quantifying network performance 

• Verifying the quality of network service 

• Attribution of network usage to users 

Unfortunately, today there is no common view on how to 
estimate the connection quality, and how to find bottlenecks 
in networks. 

Barakat et al [5] propose a model that relies on flow-level 
information to compute the total (aggregate) rate of data 
observed on an IP backbone link. For modelling purposes, 
the traffic is viewed as the superposition (i.e. multiplexing) 
of a large number of flows that arrive at random times and 
that stay active for random periods. 

Our paper presents a technique for estimating the network 
behaviour based on the utilization curve which is represent- 
ing correlation between link utilization and the number of 
active flows in it. We implicitly argue that the number of 
active flows may be considered as the real network state [5J 
[Pf] and is consequently, a better indication of utilization or 
desired operating point. 

Our objective is to gather knowledge and plot the curve for 
network quality and related terms for utilization such as: 
length of operational region, mean flow performance, confi- 
dence interval, points of overload, etc. Traffic measurement 
and analysis is extended to consider the network quality for 
a large network or for a high-speed backbone [10| 1 14] . This 
will help discover possible bottlenecks on external links of 
enterprize network or backbones [9]. We locate the threshold 
point at which the addition of new flows does not increase 
the link utilization. 

In order to prove our hypothesis we took measurements 
from the border gateway routers of Russian Internet Service 
Provider FREEnet. FREEnet (The Network For Research, 
Education and Engineering), an academic and research net- 
work, was founded on July 20, 1991, by the N.D.Zelinsky 
Institute of Organic Chemistry (IOC RAS) at the Center 
of Computer Assistance for Chemical Research (CACR). It 
assists and fulfills the networking needs of the research in- 
stitutes of Russian Academy of Sciences, universities, col- 



leges, and other research and academic institutions of Rus- 
sia. FREEnet provides IPv4/IPv6 connectivity, worldwide 
multicast IP connectivity, DNS services, stratum 1 time ser- 
vice, mail relaying, collocation and hosting services, assorted 
information services. 

Total capacity of upstream links is presently 2.1 Gbps, 1.3 
Gbps links interconnect FREEnet with its peers (excluding 
FREEnet members). There are 6 regional branches that 
operate independently but they are cooperating as peers 
in accordance with FREEnet Charter principles. Tens of 
thousands researchers from hundreds institutions country- 
wide enjoy FREEnet services. 

Earlier, in order to prove our hypothesis we took measure- 
ments from the border gateway routers of Russian ISP "Sama- 
raTelecom" (ST) and from HEAnet - Ireland's National Re- 
search and Education Network. 

In the course of experiment average flow performance in 
investigated networks considerably differs. So flow perfor- 
mance in provincial Russian networks more then one order 
less similar indicator HEAnet. These results are completely 
not surprising; essentially they reflect quality of network ser- 
vices. At once there was an idea to apply the model found 
us for the comparative analysis of large networks or date- 
centers. Thus it is possible to use unique parameter - mean 
flow performance. 

This paper describes an apllication of simple flow-based model 
[2J [3] to identify the quality of the connection and the in- 
stance when a backbone upgrade is required. We present 
our findings under the following headings: 

• Section [2]- using queueing theory for flow-based anal- 
ysis of a backbone 

• Section[3]discusses three states of network on flow level 

• Section [4]- a test for network quality 

• Section [5] - results from experiments conducted in the 
real ISP's 

• Section[6]describes techniques of comparison of a com- 
munication quality in the large networks. 

2. THE FLOW BASED MODEL 

In this paper we present traffic as a stationary process, us- 
ing the results from the papers of Barakat et al [5] and Ben 
Fredj et al [3]. They proposed a traffic model for uncon- 
gested backbone links that is simple enough to be used in 
network operations and engineering. The model of Barakat 
et al relies on Poisson shot-noise. With only 3 parameters 
(A, arrival rate of flows, E[S n ], average size of a flow, and 
Ei[Sn/D n ], average value for the ratio of the square of a flow 
size and its duration), the model provides approximations 
for the average of the total rate (the throughput) on a back- 
bone link and for its variations at short timescales. The 
model is designed to be general so that it can be easily used 
without any constraints from the definition of flows, or on 
the application or the transport protocol. In summary, this 
model allows us to completely characterize the data rate on 
a backbone link based on the following inputs: 



• Session arrivals in any period where the traffic inten- 
sity is approximately constant are accurately modelled 
by a homogeneous Poisson process of finite rate A. The 
measurements of Barakat et al [3] showed that the 
arrival rate A remains pretty constant for at least a 
30 minute interval. In general, this assumption can 
be relaxed to more general processes such as MAPs 
(Markov Arrival Processes) \U, or non homogeneous 
Poisson processes, but we will keep working with it for 
simplicity of the analysis. 

• The distribution of flow sizes S n and flow durations 
D n . In this paper we denote T n as the arrival time of 
the n-th flow, S„ as its size (e.g., in bits), and D n as its 
duration (e.g., in seconds). Sequences S n and D n also 
form independently of each other and are identically 
distributed sequences. 

• The flow rate function (shot) is X n (t). A flow is called 
active at time t when T„ < t < T n + D n . Define as 
X n (t — T n ) the rate of the n-th flow at time t (e.g., in 
bits/s), with X n (t — T n ) equal to zero for t < T n and 
for t>T n + D n . 

Define B(t) as the total rate of data (e.g., in bits/s) on the 
modeled link at time t. It is determined by adding the rates 
of the different flows. We can then write 

B(t)=^X n (t-T n ) (1) 

The process from Eq. |TJ can describe the number of active 
flows N found at time t in an M/G/oo queue [12] , if X„(t — 
T n ) = latte [T n ,Tn + D n ]. 

The model presented by Barakat et al [3] can compute the 
average and the variation of traffic on the backbone. In 
summary: 

• The average total rate of the traffic is given by the two 
parameters A and E^]: 

E[B(t)} = XE[S n ] (2) 

• The variance of the total rate Y[B(t)] (i.e., burstiness 
of the traffic) is given by the two parameters A and 
nSl/Dn]: 

V[B(t)} = XE[S n /D n ] (3) 

It should be mentioned that Eq. |2| is true only for the ideal 
case of a backbone link of unrestricted capacity, that can be 
applied to underloaded links. The main drawback of the 
ratio |2| is its lack of definite usage limits, due to the fact 
that variables A^fSVi] describing the system are in no way 
connected with its current state. The average flow size 
does not depend on a specific system, it is an universal value 
determined by the current distribution of file sizes found in 
the Internet. 

The arrival rate of flows A describes the user' s behaviour 
and doesn' t depend on the network state and utilization. 
The cumulative number of flows that arrive at a link will 
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Figure 1: Link utilization vs the number of active 
flows 

remain linear even if the network has problems and doesn' t 
satisfy all the incoming demands. 

In order to describe the real network state with arbitrary 
load we should use Little' s law: 

N = \E[D n ] (4) 

Here E[D n ] is the mean duration of flow and N is the mean 
number of active flows. Formula Q is true [12] for any flow 
duration and thus for an arbitrary flow size distribution and 
rate limit. This formula describes the network state more 
precisely than Eq. |2| as the average number of active flows 
on the bandwidth unit increases with the utilization. In 
other words, the average duration of flow E[D n ] enables us 
to judge the real network state in contrast to its average 
value E[S„]. 

3. NETWORK STATES ON FLOW LEVEL 

In order to analyze the connection quality at the backbone 
area or the link to the provider we are going to investigate a 
graphical dependence between the link utilization U and the 
number of active flows N in it [5] . These variables are easy 
measurable quantities in despite of average values E[D„] and 
E[Sn] • The separate network state is pictured by single point 
on coordinate plane with axes N and U. The curve depicting 
average values has been shown in Fig. [T] 

On the curve shown in Fig. [I] three parts can be identified, 
corresponding to the different network states. The first part 
of the curve describes the network state close to the ideal. If 
the investigated link has unrestricted capacity there should 
be a stable linear relationship between number of lows N and 
link utilization U. The curve describing the network behav- 
ior beyond a certain point will be convex. The linear part 
of the curve corresponding to the ideal network behavior is 
defined as the operational region. The operational region 
ends at the threshold point which should be found exper- 
imentally. The dislocation of this point depends on many 
factors, such as, transport layer protocol, network topology, 
the amount of buffering at the link, etc. 

The second part of the curve corresponds to the moderately 
loaded network, when the diversion from the ideal network 
state becomes obvious. There is an increase in the average 



duration of a flow compared to the working area, and there- 
fore, a larger number of active flows on the bandwidth unit 
characteristic of this network state. 

The third part of the curve corresponds to the totally dis- 
abled network with considerable packet loss evident. We 
propose some simple preliminary models for an overloaded 
link, accounting for user impatience and reattempt behaviour. 
In a real network, if demand exceeds capacity, the number of 
flows in progress does not increase indefinitely. As perflow 
throughput decreases, some flows or sessions will be inter- 
rupted, due either to user impatience or to aborts by TCP 
or higher layer protocols. 

In the end of this section estimation of confidence interval is 
given for the operational region of our curve. Since the total 
rate is the result of multiplexing of N(t) flows of indepen- 
dent rates, the Central Limit Theorem [12] tells us that the 
distribution of B(t) tends to Gaussian at high loads, which 
is typical of backbone links. As it is mentioned in Section[2] 
the variance of the total rate requires two parameters: the 
arrival rate of flows A and the expectation of the ratio be- 
tween the square of the size of a flow duration K[Sn/D n ] 
(see Eq.|3|. It tells us that the total rate should lie between 
E[B]-A{£)y/V(Bj and ~E[B] + A(e) y/V(B) in order to pro- 
vide a required quality of service. When we talked about 
the required quality of service we implied the accordance of 
network behavior to Eq. (J2|, here A{e) is the e-quantile of 
the centered and normalized total rate B(t). 

Taking into account Eqs. |2]|4| and theorems about aver- 
age values the confidence interval of the bandwidth B on a 
operational region of our curve is 

B — b(N ± aA(e)VN), (5) 

here b = E[S n /D n ] is the average flow performance which 
characterizesis the speed of user communication, coefficient 
a could be found experimentally. 

4. TESTBED SETUP 

In order to prove our hypothesis we took measurements on 
border gateway routers from FREEnet, HEAnet - Ireland's 
National Research and Education Network, and also from 
the Russian ISP "SamaraTelecom" (ST). All networks have 
several internal and external links. ST's basic load lies on 
one channel to the Internet, whereas HEAnet and FREEnet 
relies on a number of connections. Measurements from Gi- 
gabit links were taken for FREEnet and 155Mbps, 622Mbps 
are for HEAnet and ST. The utilization of these links varies 
widely from 5% to 60% with a clearly identifiable busy pe- 
riod. 

A passive monitoring system based on Cisco's NetFlow [7] 
technology was used to collect link utilization values and ac- 
tive flow numbers in real-time. In Moscow and Samara we 
measured on a Cisco 7206 router with NetFlow switched on. 
At HEAnet a Cisco 12008 was utilized. A detailed descrip- 
tion of Cisco NetFlow can be found in the Cisco documen- 
tation [7]. 

This is achieved using the following commands on the Cisco 
7206: 



• sh ip cache flow - gives information about the number 
of active and inactive flows, about the parameters of 
the flows in the real time. 

• show interface summary - gives information about the 
current link utilization. 

On a GSR Router these commands look like: 

• enable 

• attach slot-number 

• show ip cache flow 

The FREEnet data was obtained using scripts running every 
30 minutes from middle of January to the end of March 
2008. The data sets from two routers of FREEnet have 
been collected for further analysis. The full loading of first 
router varies in limits of hundreds megabits per second (100- 
220 Mbps) and tens megabits for second router (see Fig.[5|. 
During the tests we fixed information about any network 
events that could influence on connection quality. The ST 
data were recorded at 30-minute intervals, twenty-four hours 
a day for a week to discover network behavior with different 
loading levels. The HEAnet data was obtained using scripts 
running every 5 minutes for a period of 72 hours. It is quite 
easy to write a script, which will collect the data from the 
router to the management server. 

5. EXPERIMENTAL VALIDATION AND DI- 
AGNOSTIC OF TROUBLESHOOTING 

The network state at every instant describes by point on two- 
dimensional plot where abscise shows the number of active 
flow iV and ordinate shows router loading B. Basic tasks of 
experimental validation of our model consist in 

• construction of curve of average values and its com- 
parison with theoretical prediction showed on Fig. [T] 

• calculation of variance for flow performances and ver- 
ification of parabolic form of confidence interval from 
Eq. {5) 

• examination on normal distribution for flow perfor- 
mances 

In order to do experimental test of our model the data set 
should be divided into several intervals depending on num- 
ber of active flows. Inside each interval the average values 
and their variance were calculated for flow performances as 
well as for other parameters characterizing network states. 

The earlier tests have been conducted on the boundary router 
of the ST and HEAnet, and they didn't allow to verify the 
model with high precision. The new data from FREEnet 
contains thousands of points describing network states. 

FREEnet Data Set 1 has been divided into seven intervals 
according to the number of active flows (15000 - 20000, 
20000 - 25000, 25000 - 30000, 30000 - 40000, 50000 - 60000, 



Table 1: Parameters for active flows of FREEnet, 
Data Set 1, 2008 



n 


A7 


is, 


a{B), 


o, 


a(b), 






ivi ops 


1V1 Opb 


bps 


bps 


1 


17489 


113.1 


23.1 


6784 


1386 


2 


23260 


126.0 


21.4 


5682 


965 


3 


27007 


152.0 


39.2 


5628 


1452 


4 


34902 


156.7 


26.9 


4990 


770 


5 


45104 


163.9 


33.9 


3634 


752 


6 


55019 


176.3 


33.2 


3205 


604 


7 


64778 


215.4 


42.2 


3325 


652 




working 
area of 
network 



10000 2 0000 30000 40000 5000D 60000 



N 




10000 20000 30000 40000 50000 G0000 /V 



Figure 2: FREEnet router loading vs the number of 
active flows, DataSetl 



Figure 3: The operational region of network, 
DataSetl 



Table 2: Parameters for active flows of FREEnet, 
Data Set 2, 2008 



n 


TV 


B, 


a(B), 


6, 


(7(6), 






Mbps 


Mbps 


bps 


bps 


1 


5446 


15.42 


2.25 


2843 


413 


2 


6531 


17.11 


2.45 


2364 


377 


3 


7508 


17.74 


2.35 


2364 


313 


4 


8370 


18.92 


2.24 


2261 


268 


5 


9443 


20.67 


3.81 


2190 


404 


6 


15495 


28.05 


5.40 


1811 


349 



> 60000). Inside each interval basic parameters characteriz- 
ing active flows have been calculated and the result is rep- 
resented in the Table [I] Here TV is the average number of 
active flows for the interval denoted by n. B describes the 
average router loading in Megabit per second (Mbps), and b 
is average flow performance measured in bit per second (bps). 
a(b),a(B) are the standard deviation for flow performances 
b and router loading B correspondingly. 

The results of the measurement for FREEnet Data Set 1 
are pictured on Fig. [2] Basic curve is constructed as the line 
of average values, it describes network states on flow level. 
The error bar restricts the confidential interval for network 
states. Comparison with theoretical prediction from Sec- 
tion |3] leads to the conclusion that the area of network ex- 
ploitation lies inside the operational region. 

In order to restrict the operational region the straight por- 
tion of curve from Fig. [2] should be marked as it is shown on 
Fig. [3] The slope angle of this straight line is found as the 
average flow performance b. Only three initial points from 
investigated data set may be placed in the limits of opera- 
tional region. The angle of inclination gives the average flow 
performance equal to 5700 bps for FREEnet. If the number 
of flows exceeds 30000 then network gets moderately load- 
ing that leads to a reduction of the flow performance. The 
router loading does not increase uniformly with the num- 
ber of requests, and the connection quality becomes almost 
twice as bad (see Table [IJ. 

In Table[2]the Data Set 2 from the second router of FREEnet 
with lower loading is presented. Investigated region divides 
into six intervals according to the number of active flows 



(5000 - 6000, 6000 - 7000, 7000 - 8000, 9000 - 10000, > 10000) . 
Operational region for second router of FREEnet is restricted 
by 10 000 active flows as it is shown in Fig. [4] Only last inter- 
vals should be excluded from straight portion of utilization 
curve. 

Fig. [5] illustrates the network states and the form of con- 
fidence interval for operation region with normal quantile 
function A{e) = 0.05. 

A correlation coefficient indicates the strength and direc- 
tion of a linear relationship between two random variables. 
In order to verify the parabolic form of confidence interval 
the correlation coefficient between variables o~i (B) and s/Nl 
should be calculated for both DataSet of FREEnet (see Ta- 
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Figure 4: The operational region of network, 
DataSet2 
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Figure 5: Confidence interval for operation region, 
DataSet2 from FREEnet 



Table 3: Statistical tests, Data Set 1, 2008 
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bles[l]and[2|. Comparing second and forth columns of men- 
tioned Tables the values of correlation coefficient are equal 
0.70 for DataSetl and 0.93 for DataSet2. These magnitudes 
allow us to say about high correlation between theoretical 
model and its experimental examination. 

A significant question concerns the numerical value for the 
the numerical coefficient cti,at2 from Eq. dEl. The function 
A(e) can be computed using the Gaussian approximation, 
which gives for example A(0.05) = 1.96. Our data from 
Tables [l] [2] allow calculating their magnitudes: 

ai^l3,a 2 ~4.5 (6) 

Significant assumption underlies a theoretical model that 
distribution of flow performances 6 may be considered as 
Gaussian distribution. The number of testing network states 
inside many intervals from Tables [T] [2] allow to check a given 
set for similarity to the normal distribution. Here we use 
the Pearson \ 2 test and the results of this test could be 
found in Tables [3] and [4] Column 2 shows the value of \ 2 
for a — 0.95, in round brackets it is shown Table values 
for x 2 ■ ft should be noted that all investigated intervals 
with sufficient number of states discover the normal type of 
distribution. 

In conclusion of this Section it should be noted that expres- 
sion |5| allows formulating the easy rule how to display the 
network defects. If two consistent measurements running ev- 
ery 5(30) minutes give the deviation of real network states 
Bi,Ni from the confidence interval with A(0.05) then net- 
work problem has been detected. Confidence interval is de- 
scribed by flow performance b and the values a, u(B) which 



Table 4: Statistical tests, Data Set 2, 2008 
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Figure 6: Detection of anomaly network state 

may be found only as result of data processing. Unfortu- 
nately, the corresponding software is not completed yet. 

This rule received an apt illustration during FREEnet net- 
work testing. A network incident has been detected: a wide 
links to large FTP server has been temporally turned down. 
These anomaly network states Bi,Ni departure the confi- 
dence interval corresponding to standard behavior of inves- 
tigated network and form separate cluster as it has shown 
on Fig. [6] 

So our model receives the experimental confirmation and 
the diagnostic method based on introduction of confidence 
interval may be applied to network monitoring. 

6. PARAMETERS OF COMPARATIVE STUD- 
IES 

During experiments it has been noticed that the average 
flow performances in investigated networks are considerable 
different. 

For different Russian networks this parameter varies from 
hundreds bps to tens Kbps, for regional network it exceeds 
2Kbps very seldom. For comparison, the mean flow perfor- 
mance in HEAnet differs on almost one order and equals 
15000 bps that allows them to provide high speed video ap- 
plications through public network. In other words the in- 
vestigated Russian network needs expanding bandwidth of 
trunk links, especially those ones that connect remote re- 
gions. It is problem number one for Russian Networks for 
Science and Education. 

The idea evolved to apply our model for comparative analy- 
sis of large networks or data-center connectivity. The equa- 
tion |5f allows us to compare the length of operation regions, 
mean flow performances, the width of confidence intervals, 
etc. 



It should be noted that the curve of mean values from Fig. [2] 
describes the behavior of investigated network as a whole. 
Mean flow performance b allows to judge about the quality 
of network services provided to single user. The calculation 
of active flow number has a significant feature; the flow is 
considered active for a long time after transmission of the 
last packet. Therefore the real speed of network services 
exceeds the value b almost by one order. 

From the end user's point of view the mean flow performance 
is the basic universal index. Principally, this value should 
be considered as the basic index characterizing the quality 
of network services. 

There is a dependence of access to high-speed Internet ser- 
vices and mean flow performance. For example, streaming 
video began possible to look at our university as soon as 
the given indicator has exceeded value in 10 Kbps. It is 
very much a rough estimate, additional researches are nec- 
essary to find boundary values of mean flow performance, 
necessary for introduction of this or that high-speed service. 
This problem in networks of cellular operators where stan- 
dards of data transmission GPRS, EDGE and even 3G are 
not capable to give yet high speeds to an unlimited circle of 
clients is especially actual. 

7. CONCLUSION 

In this paper some methods are described that allow us to 
evaluate connection quality in large networks based on flow 
technology, and give an indication as to when capacity needs 
to be increased. At the moment we are working on devel- 
oping utilities, which will make it possible to construct the 
dependence of link loading on the number of active flows au- 
tomatically, and calculate the length of the operation region 
as well as coefficients for confidence interval. 

The special attention has been given how by means of the 
constructed model to make a technique of comparison of the 
big networks. It has been established that as object for com- 
parison the unique parameter - mean flow performance can 
act. The high-grade comparative analysis should contain the 
conformity table between mean flow performance and possi- 
bility of start high-speed Internet services that demands the 
further experiments. 

This paper has demonstrated that it is possible to easily 
determine the confidence interval, operation region and the 
overload point of a network connection, utilizing low cost 
commodity hardware and simple software. This allows us 
to identify the anomaly network states and moment when 
a backbone upgrade is required. Further experiments are 
necessary in order to develop software utilities for this pur- 
pose. Thus, providing analytical generalizations, we estab- 
lished common terminology for processes, taking place in 
networks. 
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